Learning with Limited Visibility

This paper surveys recent studies of learning problems in which the learner faces restrictions on the amount of information he can extract from each example he encounters. Our main framework for the analysis of such scenarios is the RFA (Restricted Focus of Attention) model. While being a natural reenement of the PAC learning model, some of the fundamental PAC-learning results and techniques fail in the RFA paradigm; learnability in the RFA model is no longer characterized by the VC dimension, and many PAC learning algorithms are not applicable in the RFA setting. Hence, the RFA formulation reeects the need for new techniques and tools to cope with some fundamental constraints of realistic learning problems. We also present some paradigms and algorithms that may serve as a rst step towards answering this need. Two main types of restrictions can be considered in the general RFA setting: In the more stringent one, called k-RFA, only k of the n attributes of each example are revealed to the learner, while in the more permissive one, called k-wRFA, the restriction is made on the size of each observation (k bits), and no restriction is made on how the observations are extracted from the examples. We show an information-theoretic characterization of RFA learnability upon which we build a general tool for proving hardness results. We then apply this and other new techniques for studying RFA learning to two particularly expressive function classes, k-decision-lists (k-DL) and k-TOP, the class of thresholds of parity functions in which each parity function takes at most k inputs. Among other results, we show a hardness result for k-RFA learnability of k-DL, k n?2. In sharp contrast, an (n ? 1)-RFA algorithm for learning (n ? 1)-DL is presented. Similarly, we prove that 1-DL is learnable if and only if at least half of the inputs are visible in each instance. In addition, we show that there is a uniform-distribution k-RFA learning algorithm for the class of k-DL. For k-TOP we show weak learnability by a k-RFA algorithm (with eecient time and sample complexity for constant k) and strong uniform-distribution k-RFA learnability of k-TOP with eecient sample complexity for constant k. Finally, by combining some of our k-DL and k-TOP results, we show that, unlike the PAC model, weak learning does not imply strong learning in the k-RFA model. We also show a general technique for composing eecient k-RFA algorithms, and …